FOUNDATION MODELS

LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2024.naacl-demo)

Challenge:	This tutorial explores the challenges of continual learning in large language models . participants will learn strategies to mitigate forgetting and manage data and evaluation pipelines .
Approach:	This tutorial offers a comprehensive exploration of continual learning in the context of large language models.
Outcome:	This tutorial explores the challenges of continual learning in large language models . participants will learn how to manage data and evaluation pipelines and adapt responsibly .

Copied to clipboard

Shizhe Diao, Rui Pan, Hanze Dong, KaShun Shum, Jipeng Zhang, Wei Xiong, Tong Zhang

SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognition (2025.emnlp-main)

Challenge:	Foundation models have demonstrated a great ability to achieve general human-level intelligence far beyond traditional approaches.
Approach:	They propose a toolkit to simplify the finetuning of general foundation models.
Outcome:	The proposed toolkit simplifies the domain- and task-aware finetuning of general foundation models with limited computing resources.

Copied to clipboard

Zechen Li, Shohreh Deldari, Linyao Chen, Hao Xue, Flora D. Salim

OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models (2024.findings-naacl)

Challenge:	SensorLLM is a timeseries classification framework that can perform human activity recognition tasks.
Approach:	They propose a framework that enables Large Language Models to perform human activity recognition from sensor time-series data.
Outcome:	The proposed framework can perform human activity recognition (HAR) tasks with human inputs.

Copied to clipboard

Yuxuan Kuang, Hai Lin, Meng Jiang

SciSketch: An Open-source Framework for Automated Schematic Diagram Generation in Scientific Papers (2025.emnlp-demos)

Challenge:	Existing methods for object navigation are limited to household datasets with close-set objects, and they lack the ability to generalize to new environments in a zero-shot manner.
Approach:	They propose a framework that leverages reasoning abilities of large vision language models to extract proposed objects from natural language instructions that meet the user’s demand.
Outcome:	The proposed framework surpasses baselines on all metrics and can be used in a HM3D ObjectNav benchmark.

Copied to clipboard

Zihang Wang, Yilun Zhao, Kaiyan Zhang, Chen Zhao, Manasi Patwardhan, Arman Cohan

MEAL: Stable and Active Learning for Few-Shot Prompting (2023.findings-emnlp)

Challenge:	SCISKETCH is an open-source framework that supports two automated workflows for schematic diagram generation using foundation models.
Approach:	They propose an open-source framework that supports two automated workflows for schematic diagram generation using foundation models.
Outcome:	The open-source framework outperforms several state-of-the-art foundation models in generating schematic diagrams for scientific papers.

Copied to clipboard

Abdullatif Köksal, Timo Schick, Hinrich Schuetze

Low-Resource NLP

Challenge:	Existing methods for few-shot classification have high variance across different sets of few shots and finetuning runs.
Approach:	They propose novel ensembling methods that significantly reduce run variability and introduce a new active learning criterion for data selection.
Outcome:	The proposed method significantly reduces run variability and improves performance on five tasks.

Pragmatic Reasoning Unlocks Quantifier Semantics for Foundation Models (2023.emnlp-main)

Copied to clipboard

Yiyuan Li, Rakesh Menon, Sayan Ghosh, Shashank Srivastava

SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Data (2023.findings-emnlp)

Challenge:	Generalized quantifiers are used to indicate the proportions predicates satisfy (e.g., some apples are red).
Approach:	They propose a framework to model quantifier semantics for textbased foundation models by combining natural language inference and the Rational Speech Acts framework.
Outcome:	The proposed framework shows a 20% improvement over a literal listener baseline in predicting percentage scopes for quantifier comprehension even with no training.

Copied to clipboard

Ruoxi Sun, Sercan Arik, Rajarishi Sinha, Hootan Nakhost, Hanjun Dai, Pengcheng Yin, Tomas Pfister

Programming Languages in NLP

Challenge:	Text-to-SQL aims to automate the process of generating SQL queries on a database from natural language text.
Approach:	They propose a method to improve few-shot prompting capabilities of Text-to-SQL for Large Language Models (LLMs) they propose 'SQlPrompt' which aims to diversify the SQL proposals during consistency selection with different prompt designs and foundation models.
Outcome:	The proposed method outperforms previous approaches for in-context learning with zero labeled data by a large margin, closing the gap with finetuning state-of-the-art with thousands of labeles.

Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks (2023.findings-emnlp)

Copied to clipboard

Xinsong Zhang, Yan Zeng, Jipeng Zhang, Hang Li

MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment (2026.acl-industry)

Challenge:	Existing foundation models can only perform the best in one type of understanding tasks.
Approach:	They propose a method for training a general foundation model, X-FM, using text, image, and image-text data.
Outcome:	The proposed method outperforms existing foundation models on language, vision, and vision-language understanding tasks.

Copied to clipboard

Hanxian Huang, Igor Fedorov, Andrey Gromov, Bernard Beckerman, Naveen Suda, David Eriksson, Maximilian Balandat, Rylan Conway, Patrick Huber, Chinnadhurai Sankar, Ayushi Dalmia, Zechun Liu, Lemeng Wu, Tarek Elgamal, Adithya Sagar, Vikas Chandra, Raghuraman Krishnamoorthi

Green & Sustainable NLP

Challenge:	MobileLLM-Flash is a family of foundation models for efficient on-device use with strong capabilities.
Approach:	They propose a method for designing on-device large language models under mobile latency constraints using hardware-in-the-loop architecture search.
Outcome:	The proposed model is amenable to industry-scale deployment and is compatible with mobile runtimes like Executorch.

KWJA: A Unified Japanese Analyzer Based on Foundation Models (2023.acl-demo)

Copied to clipboard

Nobuhiro Ueda, Kazumasa Omura, Takashi Kodama, Hirokazu Kiyomaru, Yugo Murawaki, Daisuke Kawahara, Sadao Kurohashi

Challenge:	KWJA supports a wide range of tasks including typo correction, word segmentation, word normalization, named entity recognition, dependency parsing, PAS analysis, bridging reference resolution, coreference resolution, and discourse relation analysis.
Approach:	They propose to build a Japanese text analyzer based on foundation models that performs a wide range of tasks.
Outcome:	The proposed model performs better in a multi-task manner than other analyzers with specialized models.

Enhancing Foundation Models in Transaction Understanding with LLM-based Sentence Embeddings (2025.emnlp-industry)

Copied to clipboard

Xiran Fan, Zhimeng Jiang, Chin-Chia Michael Yeh, Yuzhong Chen, Yingtong Dou, Menghai Pan, Yan Zheng

ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion (2025.findings-emnlp)

Challenge:	Existing foundation models for tabular transactional data rely on index-based representations for categorical merchant fields.
Approach:	They propose a framework that uses LLM-generated embeddings as semantic initializations for lightweight transaction models.
Outcome:	The proposed framework improves performance on large transaction datasets.

Copied to clipboard

Rana Shahroz, Dongwen Tang, Pingzhi Li, Kai Wang, Tianlong Chen

Domain Adaptation of Foundation LLMs for e-Commerce (2025.acl-industry)

Challenge:	Existing approaches to low-rank Adaptation (LoRA) are limited in scalability and controllability.
Approach:	They propose a conditional recurrent diffusion framework that generates LoRA parameters directly . they integrate model architecture and textual task specifications to generate task-specific parameters .
Outcome:	The proposed framework scales to billions-of-parameter LLMs and maintains controllability.

Copied to clipboard

Christian Herold, Michael Kozielski, Tala Bazazo, Pavel Petrushkov, Yannick Versley, Seyyed Hadi Hashemi, Patrycja Cieplicka, Dominika Basaj, Shahram Khadivi

Model Balancing Helps Low-data Training and Fine-tuning (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have greatly improved the performance on most natural language tasks, and often show surprisingly good zero-shot generalization to new domains.
Approach:	They propose to continuously pretrain the Llama 3.1 base models on 1 trillion tokens of e-commerce data to introduce domain specific knowledge into the model while at the same time keeping the general capabilities intact.
Outcome:	The proposed model can be adapted to the new domain without sacrificing performance on general domain tasks.

Copied to clipboard

Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, Yaoqing Yang

Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (2024.acl-long)

Challenge:	Recent advances in foundation models have emphasized the need to align pre-trained models with specialized domains using small, curated datasets.
Approach:	They propose a layer-wise learning rate scheduler that balances training quality across layers . they adapt it to a curated dataset to achieve alignment with specialized domains .
Outcome:	The proposed model shows that it can be used to balance training quality across layers and improve low-data training and fine-tuning for both NLP and SciML tasks.

Copied to clipboard

Baizhou Huang, Shuai Lu, Xiaojun Wan, Nan Duan

Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Models (2025.findings-naacl)

Challenge:	Existing work utilizes verification properties to verify and re-rank solutions in a majority voting manner, but this assumption may not hold.
Approach:	They propose a multi-perspective self-consistency framework that incorporates both inter- and intra-consistency across outputs from multiple perspectives.
Outcome:	The proposed framework significantly boosts performance of foundation models on various benchmarks, including HumanEval (+15.91%), MBPP (+6.43%) and CodeContests (+9.37%).

Copied to clipboard

Aashiq Muhamed, Mona T. Diab, Virginia Smith

Composite Backdoor Attacks Against Large Language Models (2024.findings-naacl)

Challenge:	Sparse Autoencoders (SAEs) are a promising tool for disentangling FM representations, but they struggle to capture rare, yet crucial concepts in the data.
Approach:	They propose a technique to train Sparse Autoencoders to illuminate elusive dark matter features by focusing on specific subdomains.
Outcome:	The proposed method achieves 12.5% better classification accuracy than general-purpose SAEs when applied to remove spurious gender information.

Copied to clipboard

Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, Yang Zhang

Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Cost (2024.lrec-main)

Challenge:	Large language models (LLMs) have demonstrated superior performance on various tasks, but untrustworthy third-party LLMs may covertly introduce vulnerabilities for downstream tasks.
Approach:	They propose a composite backdoor attack that scatters multiple trigger keys in different prompt components.
Outcome:	The proposed attack achieves 100% Attack Success Rate (ASR) with a False Triggered Rate (FTR) below 2.06% and negligible model accuracy degradation.

Copied to clipboard

Oana Ignat, Longju Bai, Joan C. Nwatu, Rada Mihalcea

Visual Data in NLP

Challenge:	Current foundation models have shown impressive performance across various tasks, but they are not effective for everyone due to the imbalanced geographical and economic representation of the data used in the training process.
Approach:	They propose to identify the data to be annotated to balance model performance and annotation costs by finding countries with visual similarity for the topics.
Outcome:	The proposed methods improve model performance and reduce annotation costs by using data from countries with higher visual similarity for these topics.

Tag-grounded Visual Instruction Tuning with Retrieval Augmentation (2024.emnlp-main)

Copied to clipboard

Daiqing Qi, Handong Zhao, Zijun Wei, Sheng Li

PEFTDebias : Capturing debiasing information using PEFTs (2023.emnlp-main)

Challenge:	Multimodal Large Language Models (MLLMs) have seen remarkable progress in providing general instruction-following ability, but struggle with critical problems when required to provide a detailed and accurate response to a visual instruction.
Approach:	They propose to enhance the mapping process by using retrieval-augmented tag tokens, which contain rich object-aware information such as object names and attributes.
Outcome:	The proposed model outperforms baselines that share the same language model and training data on 12 benchmarks and shows zero-shot capability when provided with specific datastores.

Copied to clipboard

Sumit Agarwal, Aditya Veerubhotla, Srijan Bansal

Robustness in NLP

Challenge:	Recent research suggests that biases acquired during pretraining can propagate to downstream models, resulting in superficial text dependencies and potential implicit bias.
Approach:	They propose a parameter-efficient fine-tuning approach to mitigate implicit biases within foundation models by incorporating parameters into the model and freezing them during the fine-uning process.
Outcome:	The proposed method reduces biases in foundation models by incorporating parameters and freezing them during fine-tuning.

AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2024.findings-naacl)

Copied to clipboard

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, Nan Duan

Accurate and Nuanced Open-QA Evaluation Through Textual Entailment (2024.findings-acl)

Challenge:	Traditional benchmarks for evaluating foundation models often fail to accurately represent their general abilities for human-centric tasks.
Approach:	They propose a bilingual benchmark to assess foundation models in the context of human-centric standardized exams such as college entrance exams, law school admission tests, and math competitions.
Outcome:	The proposed benchmark exceeds the average human performance on SAT, LSAT, and math competitions with 95% accuracy and 92.5% on the Chinese college entrance English exam.

Copied to clipboard

Peiran Yao, Denilson Barbosa

Question Answering

Challenge:	Open-domain question answering (Open-QA) evaluations are criticized for the ambiguity in questions and the lack of semantic understanding in evaluators.
Approach:	They propose to examine the entailment relations of answers to identify more informative and more general system answers.
Outcome:	The proposed evaluations offer a much closer evaluation to human judgment on NaturalQuestions and TriviaQA while being learning-free.

ixi-GEN: Efficient Industrial sLLMs through Domain Adaptive Continual Pretraining (2025.emnlp-industry)

Copied to clipboard

Seonwu Kim, Yohan Na, Kihun Kim, Hanhee Cho, Geun Lim, Mintae Kim, Seongik Park, Ki Hyun Kim, Youngsub Han, Byoung-Ki Jeon

ModularMoE: Fast LLM Customization with Parameter-Sharing Mixture-of-Experts for Low-Resource Settings (2026.findings-acl)

Challenge:	Domain Adaptive Continual Pretraining (DACP) is a method to mitigate performance degradation in small LLMs and enhance their effectiveness in target domains.
Approach:	They propose a continual pretraining methodology that optimizes sLLMs within service domains and enhances their effectiveness in target domains.
Outcome:	The proposed model achieves significant gains in target-domain performance while preserving general capabilities, offering a cost-efficient and scalable solution for enterprise-level deployment.

Copied to clipboard

Jiaxing Liu, Qi Qi, Haifeng Sun, Dunjun Li, Zirui Zhuang, Bo He, Xiang Yang, Cong Liu, Jianxin Liao, Jingyu Wang

Making Language Models Better Tool Learners with Execution Feedback (2024.naacl-long)

Challenge:	Large Language Models impose significant computational and storage burdens on personal devices . existing customization approaches incur excessive computational costs or lead to suboptimal performance .
Approach:	They propose a training framework that converts pre-trained LLMs into parameter-sharing MoE models for lightweight deployment.
Outcome:	The proposed training framework outperforms state-of-the-art training frameworks at the same sparsity level while delivering up to 2.71 inference speedup.

Copied to clipboard

Shuofei Qiao, Honghao Gui, Chengfei Lv, Qianghuai Jia, Huajun Chen, Ningyu Zhang

What Really Matters for Table LLMs? A Meta-Evaluation of Model and Data Effects (2026.findings-eacl)

Challenge:	Existing tool learning methodologies induce large language models to utilize tools indiscriminately . Existing frameworks that teach language models when and how to use tools propagate errors rather than enhance performance.
Approach:	They propose a framework that enables large language models to continually learn through feedback derived from tool execution.
Outcome:	The proposed framework can make large language models selectively use tools . it improves accuracy while enhancing insufficient tool learning, it shows .

Copied to clipboard

Naihao Deng, Sheng Zhang, Henghui Zhu, Shuaichen Chang, Jiani Zhang, Alexander Hanbo Li, Chung-Wei Hang, Hideo Kobayashi, Yiqun Hu, Patrick Ng

Unsupervised Human Preference Learning (2024.emnlp-main)

Challenge:	a series of paradigm shifts have come with distinct characteristics and challenges associated with table modeling.
Approach:	They propose to replicate four table LLMs by instruction-tuning three foundation models on four existing datasets.
Outcome:	The results show that base model choice plays a more dominant role than training data itself.

Copied to clipboard

Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Dilek Tur

A + B: A General Generator-Reader Framework for Optimizing LLMs to Unleash Synergy Potential (2024.findings-acl)

Challenge:	Existing methods for in-context learning and parameter-efficient fine-tuning fail to capture the complexity of human preferences, especially given the small, personal datasets individuals possess.
Approach:	They propose a method that uses small parameter models as preference agents to generate natural language rules that guide a larger, pre-trained model, enabling efficient personalization.
Outcome:	The proposed method outperforms baseline personalization methods on email and article datasets and significantly outperformed existing methods.

Copied to clipboard

Wei Tang, Yixin Cao, Jiahao Ying, Bo Wang, Yuyue Zhao, Yong Liao, Peng Zhou

Information Retrieval

Challenge:	Retrieval-Augmented Generation (RAG) is an effective solution to supplement necessary knowledge to large language models.
Approach:	They propose a "generate-then-read" pipeline to replace retrieval stage with generation from the LLM itself.
Outcome:	The proposed framework outperforms single models in the base and chat versions and addresses safety and helpfulness post-adaptation challenges.

Pruning the Paradox: How CLIP’s Most Informative Heads Enhance Performance While Amplifying Bias (2025.emnlp-main)

Copied to clipboard

Avinash Madasu, Vasudev Lal, Phillip Howard

Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding (2024.acl-long)

Challenge:	Large-scale vision-language models such as CLIP have advanced state-of-the-art performance in vision tasks . however, as they gain prominence in real-world applications, their embedded social biases can be harmful . et al., 2021: 103-104.
Approach:	They propose an interpretability metric that measures how consistently attention heads align with specific concepts in CLIP-like models.
Outcome:	The proposed interpretability metric measures how consistently attention heads align with specific concepts.

Copied to clipboard

Kuo Liao, Shuang Li, Meng Zhao, Liqun Liu, Mengge Xue, Zhenyu Hu, Honglin Han, Chengguo Yin

Challenge:	Recent advances in large language models (LLMs) have yielded remarkable performance, but objective mismatch issues hinder RLHF learning.
Approach:	They propose a Reinforcement Learning framework enhanced with Label-sensitive reward to enhance LLMs' alignment and generation capabilities.
Outcome:	The proposed framework improves performance on five diverse models across eight tasks.

MiTTenS: A Dataset for Evaluating Gender Mistranslation (2024.emnlp-main)

Copied to clipboard

Kevin Robinson, Sneha Kudugunta, Romina Stella, Sunipa Dev, Jasmijn Bastings

Machine Translation

Challenge:	Existing studies on gender mistranslation in translation systems have highlighted the problem . a dataset of 26 languages is presented to measure the extent of such errors .
Approach:	They propose a dataset that measures the extent of gender mistranslation in translation systems . they use handcrafted passages that target known failure patterns and synthetically generated passages .
Outcome:	The proposed dataset covers 26 languages from a variety of language families and scripts, including several traditionally under-represented in digital resources.

ECHo: A Visio-Linguistic Dataset for Event Causality Inference via Human-Centric Reasoning (2023.findings-emnlp)

Copied to clipboard

Yuxi Xie, Guanzhen Li, Min-Yen Kan

CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated Chain-of-Thought-based Reinforced Fine-Tuning (2025.emnlp-main)

Challenge:	ECHo is a diagnostic dataset of event causality inference grounded in visio-linguistic social scenarios.
Approach:	They propose a diagnostic dataset of event causality inference grounded in visio-linguistic social scenarios.
Outcome:	The proposed framework examines the reasoning capability of current AI systems on three human-centric tasks.

Copied to clipboard

Wenqiao Zhu, Ji Liu, Rongjunchen Zhang, Haipang Wu, Yulun Zhang

LongWanjuan: Towards Systematic Measurement for Long Text Quality (2024.findings-emnlp)

Challenge:	Existing methods to improve the reasoning performance of Large Language Models (LLMs) ignore annotated Chain-of-Thought (CoT) and incorporate unstable reasoning path sampling.
Approach:	They propose a Contrastive learning with annotated CoT-based Reinforced Fine-Tuning approach to enhance the reasoning performance of Large Language Models.
Outcome:	The proposed approach exploits annotated CoT and stabilizes the fine-tuning procedure by incorporating an additional unsupervised learning signal.

Copied to clipboard

Xiaoran Liu, Kai Lv, Qipeng Guo, Hang Yan, Conghui He, Xipeng Qiu, Dahua Lin

Commonsense Knowledge Transfer for Pre-trained Language Models (2023.findings-acl)

Challenge:	Existing efforts to improve data quality have focused on deduplication and the evaluation of data diversity and difficulty.
Approach:	They propose a set of metrics to evaluate the quality of long texts by evaluating three fundamental linguistic dimensions: coherence, cohesion, and complexity.
Outcome:	The proposed model improves on long-text tasks with over 160B tokens and categorizes long texts into holistic, aggregated, and chaotic types.

Copied to clipboard

Wangchunshu Zhou, Ronan Le Bras, Yejin Choi

ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models (2024.findings-acl)

Challenge:	Recent advances in pre-trained language models have transformed the landscape of natural language processing.
Approach:	They propose a framework to transfer commonsense knowledge stored in a neural commonsensing model to a general-purpose pre-trained language model.
Outcome:	Empirical results show that the proposed framework improves the model’s performance on downstream tasks that require commonsense reasoning.

Copied to clipboard

Yanan Wu, Jie Liu, Xingyuan Bu, Jiaheng Liu, Zhanhui Zhou, Yuanxing Zhang, Chenchen Zhang, ZhiqiBai ZhiqiBai, Haibin Chen, Tiezheng Ge, Wanli Ouyang, Wenbo Su, Bo Zheng

SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification (2025.acl-long)

Challenge:	ConceptMath evaluates concept-wise mathematical reasoning of Large Language Models (LLMs) Existing benchmarks that evaluate general mathematical reasoning with an average accuracy fail to probe the fine-grained failure modes of mathematical reasoning on specific datasets.
Approach:	They introduce a bilingual, fine-grained benchmark that evaluates concept-wise mathematical reasoning of Large Language Models.
Outcome:	The proposed benchmarks evaluate concept-wise mathematical reasoning of Large Language Models with concept-based accuracies.

Copied to clipboard

Chengye Wang, Yifei Shen, Zexi Kuang, Arman Cohan, Yilun Zhao

ConlangCrafter: Constructing Languages with a Multi-Hop LLM Pipeline (2026.acl-long)

Challenge:	Existing scientific claim verification benchmarks focus on textual content alone or on verifying claims based on a single table.
Approach:	They propose to use SciVer to evaluate the ability of foundation models to verify claims within a multimodal scientific context.
Outcome:	The proposed model outperforms 21 state-of-the-art models and human experts on SciVer.

Copied to clipboard

Morris Alper, Moran Yanuka, Raja Giryes, Gasper Begus

Challenge:	Constructed languages (conlangs) have played diverse roles in art, philosophy, and international communication. foundation models have revolutionized creative generation in text, images, and beyond.
Approach:	They propose a multi-hop pipeline that decomposes language design into modular stages . they use LLMs' metalinguistic reasoning capabilities to encourage diversity .
Outcome:	The proposed pipeline decomposes language design into modular stages . it leverages LLMs’ metalinguistic reasoning capabilities to encourage diversity and self-refinement feedback to encourage consistency and typological diversity.

Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RL (2024.acl-long)

Copied to clipboard

Yunseon Choi, Sangmin Bae, Seonghyun Ban, Minchan Jeong, Chuheng Zhang, Lei Song, Li Zhao, Jiang Bian, Kee-Eung Kim

FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Models (2024.emnlp-main)

Challenge:	Prompt tuning is an important technique for directing model behaviors and eliciting desired responses.
Approach:	They propose to find optimal prompt tokens using soft Q-learning to optimize models for prompt tuning.
Outcome:	The proposed method improves on baseline prompt tuning, and the results are more natural and interpretable.

Copied to clipboard

Xiaochen Wang, Jiaqi Wang, Houping Xiao, Jinghui Chen, Fenglong Ma

On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research (2023.emnlp-main)

Challenge:	Existing foundation models are limited in access to diverse modalities and privacy regulations restrict the development of comprehensive foundation models.
Approach:	They propose a knowledge injection approach to extract and inject healthcare knowledge into medical foundation models to enhance their ability to handle multiple tasks and modalities.
Outcome:	The proposed method preserves privacy and enhances the model’s ability to handle complex medical tasks involving multiple modalities.

Copied to clipboard

Luiza Pozzobon, Beyza Ermis, Patrick Lewis, Sara Hooker

Position Really Matters: Towards a Holistic Approach for Prompt Tuning (2025.findings-naacl)

Challenge:	Perception of toxicity evolves over time and differs between geographies and cultural backgrounds.
Approach:	They propose to use a more structured approach to evaluating toxicity over time . they suggest that research that relied on automatic toxicity scores may have resulted in inaccurate results.
Outcome:	The Perspective API has been updated to reflect the changes in toxicity scores.

Copied to clipboard

Xianjun Yang, Wei Cheng, Xujiang Zhao, Wenchao Yu, Linda Ruth Petzold, Haifeng Chen

On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research (2023.acl-long)

Challenge:	Prompt tuning is effective in extracting knowledge from foundation models, but its effectiveness is uncertain.
Approach:	They propose a parametric prompt tuning strategy that dynamically determines different factors of prompts based on specific tasks or instances.
Outcome:	The proposed approach improves performance across a wide range of tasks including NLP, vision recognition, and vision-language tasks.

Copied to clipboard

Made Nindyatama Nityasya, Haryo Wibowo, Alham Fikri Aji, Genta Winata, Radityo Eko Prasojo, Phil Blunsom, Adhiguna Kuncoro

Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study (2023.emnlp-main)

Challenge:	Despite rapid recent progress, current research practices conflate different sources of model improvement without conducting proper ablation studies and principled comparisons . authors conclude with recommendations for how to encourage and incentivize this line of work .
Approach:	They critique current research practices in the field of language model pre-training . they examine the success of language models pre-trained on large amounts of data .
Outcome:	The proposed models can achieve competitive or better performance than BERT under comparable conditions.

Copied to clipboard

Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, Bryan Catanzaro

mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections (2022.emnlp-main)

Challenge:	a recent study shows that retrieval-augmented LMs can improve text generation quality and accuracy.
Approach:	They propose a model that reproduces RETRO parameters while retrieving a text corpus . they find RETRO outperforms GPT on text generation with less repetition .
Outcome:	The proposed model outperforms standard retrieval-augmented GPT and retrieval augmented GTP on text generation and accuracy tasks.

Copied to clipboard

Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, He Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, Luo Si

Visual Data in NLP

Challenge:	Existing pre-trained vision-language models suffer from inefficiency and linguistic signal overwhelmed by long visual sequences in cross-modal alignment.
Approach:	They propose a vision-language foundation model with cross-modal skip-connections that can be pre-trained end-to-end on large-scale image-text pairs with both discriminative and generative objectives.
Outcome:	The proposed model achieves state-of-the-art results on a wide range of vision-language downstream tasks, including image captioning, image-text retrieval, visual grounding and visual question answering.

Deeply Coupled Cross-Modal Prompt Learning (2023.findings-acl)

Copied to clipboard

Xuejing Liu, Wei Tang, Jinghui Lu, Rui Zhao, Zhaojun Guo, Fei Tan

RecGPT: A Foundation Model for Sequential Recommendation (2025.emnlp-main)

Challenge:	Existing prompt-tuning methods focus on language branch or learn vision-language interaction in a shallow mechanism.
Approach:	They propose a Deeply coupled Cross-modal Prompt learning method based on CLIP to facilitate the interplay between vision and language with a Cross-Modal Prompting Attention mechanism.
Outcome:	The proposed method enables the interplay between vision and language with a Cross-Modal Prompt Attention mechanism.

Copied to clipboard

Yangqin Jiang, Xubin Ren, Lianghao Xia, Da Luo, Kangyi Lin, Chao Huang

MERA: A Comprehensive LLM Evaluation in Russian (2024.acl-long)

Challenge:	Existing approaches fail in cold-start and cross-domain scenarios where new users or items lack sufficient interaction history.
Approach:	They propose a foundation model for sequential recommendation that achieves genuine zero-shot generalization capabilities by deriving item representations exclusively from textual features.
Outcome:	The proposed model achieves zero-shot generalization capabilities in cold-start and cross-domain scenarios.

Copied to clipboard

Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, Ulyana Isaeva, Katerina Kolomeytseva, Daniil Moskovskiy, Elizaveta Goncharova, Nikita Savushkin, Polina Mikhailova, Anastasia Minaeva, Denis Dimitrov, Alexander Panchenko, Sergey Markov

Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving (2025.findings-acl)

Challenge:	Recent advances in foundation models have led to the emergence of powerful Large Language Models (LLMs), which showcase unprecedented tasksolving capabilities.
Approach:	They propose a method to evaluate FMs and LMs in fixed zero- and few-shot instruction settings that can be extended to other modalities.
Outcome:	The proposed evaluation methodology includes an open-source code base and a leaderboard with a submission system.

Copied to clipboard

Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, Arman Cohan

Challenge:	a benchmark for university-level physics problem solving contains 1,297 expert-annotated problems . a proprietary model, o3-mini, achieves only 59.9% accuracy, highlighting fundamental weaknesses in scientific reasoning, conceptual understanding, and mathematical precision.
Approach:	They introduce Physics, a benchmark for university-level physics problem solving.
Outcome:	The proposed model achieves only 59.9% accuracy on the most advanced model, o3-mini . the proposed model is a powerful tool for evaluating models on advanced problems .

ProcWorld: Benchmarking Large Model Planning in Reachability-Constrained Environments (2025.emnlp-main)

Copied to clipboard

Dong Wang, Xinghang Li, Zhengshen Zhang, Jirong Liu, Xiao Ma, Hanbo Zhang, Tao Kong, Huaping Liu

Socratic Style Chain-of-Thoughts Help LLMs to be a Better Reasoner (2025.findings-acl)

Challenge:	Existing benchmarks for embodied spatial reasoning and long-term planning are non-trivial due to the combinatorial complexity of long-horizon abstract reasoning.
Approach:	They propose a large-scale benchmark for partially observable embodied spatial reasoning and long-term planning with large language models and vision language models.
Outcome:	The proposed model performs better in 16 task types, 5,000 rooms, and over 10 million evaluation trajectories with diverse data distribution.

Copied to clipboard

Jiangbo Pei, Peiyu Liu, Xin Zhao, Aidong Men, Yang Liu

VIP5: Towards Multimodal Foundation Models for Recommendation (2023.findings-emnlp)

Challenge:	Synthetic data generation is a promising approach to enhance reasoning capabilities of large language models.
Approach:	They propose a multi-agent debate framework based on the Socratic questioning strategy . they use socratic questions to deepen the thinking process and broaden it to motivate self-reflection .
Outcome:	The proposed framework outperforms existing methods on math and code generation tasks while maintaining affordable costs.

Copied to clipboard

Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu, Yongfeng Zhang

CoT-VTM: Visual-to-Music Generation with Chain-of-Thought Reasoning (2025.findings-acl)

Challenge:	Recent advances in foundation models have impeded the ability for these fields to benefit from each other’s advancements.
Approach:	They propose to use a multimodal foundation model to unify various modalities and recommendation tasks under the P5 recommendation paradigm to implement personalized prompts.
Outcome:	The proposed model will unify visual, textual, and personalization modalities under the P5 recommendation paradigm and will improve recommendation performance and efficiency.

Copied to clipboard

Xikang Guan, Zheng Gu, Jing Huo, Tianyu Ding, Yang Gao

ICC : Quantifying Image Caption Concreteness for Multimodal Dataset Curation (2024.findings-acl)

Challenge:	Existing methods for visual-to-music generation lack large-scale, high-quality visual-music paired datasets and lack of direct semantic correspondence between visuals and music.
Approach:	They propose a framework that distills Chain-of-Thought reasoning to enable visual-to-music generation without paired data.
Outcome:	The proposed framework achieves optimal performance on image-to-music and video-to music tasks.

Copied to clipboard

Moran Yanuka, Morris Alper, Hadar Averbuch-Elor, Raja Giryes

Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? (2024.emnlp-main)

Challenge:	Existing methods to curation text-image data are noisy and lack the fine-grained ability to isolate the most concrete samples that provide the strongest signal for learning in a noisy dataset.
Approach:	They propose a metric that evaluates caption text without an image reference to measure its concreteness and relevancy.
Outcome:	The proposed method detects the concreteness of captions without an image reference and correlates with human evaluation of concreteness in both single-word and caption-level texts.

Copied to clipboard

Daniel Jeong, Saurabh Garg, Zachary Lipton, Michael Oberst

CRPS: Curriculum Replay via Progressive Suffixes from Successful Trajectories for Long-Horizon LLM Agents (2026.findings-acl)

Challenge:	Several studies claim that domain-adaptive pretraining improves performance on downstream medical tasks.
Approach:	They compare medical LLMs and VLMs against their corresponding base models . they find that medical Lms outperform their base models in 12.1% of cases .
Outcome:	The proposed models outperform their base models on medical questions and tasks in 12.1% of cases and reach a tie in 49.8% of cases.

Copied to clipboard

Zijing Zhang, Xiajie Yang

Challenge:	Long-horizon agents trained with sparse terminal rewards experience slow and unstable learning . this is amplified by group-normalized on-policy objectives, which yield degenerate advantages and weak learning signals.
Approach:	They propose a lightweight RL-training strategy that turns terminal successes into within-trajectory curriculum by replaying from successful suffix states.
Outcome:	The proposed strategy outperforms full-episode GRPO and naive experience replay across ALFWorld and WebShop with different foundation models.

A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models (2024.findings-emnlp)

Copied to clipboard

Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, Aman Chadha

RealBehavior: A Framework for Faithfully Characterizing Foundation Models’ Human-like Behavior Mechanisms (2023.findings-emnlp)

Challenge:	a growing need to understand and alleviate FMs' propensity to produce hallucinated outputs, especially in high-stakes applications.
Approach:	They propose a framework for detecting and mitigating hallucination in FMs . they synthesize recent advancements in detection and mitigation techniques .
Outcome:	The proposed framework provides valuable insights for researchers, developers, and practitioners.

Copied to clipboard

Enyu Zhou, Rui Zheng, Zhiheng Xi, Songyang Gao, Xiaoran Fan, Zichu Fei, Jingting Ye, Tao Gui, Qi Zhang, Xuanjing Huang

Challenge:	Existing studies on human-like behaviors in foundation models do not verify their faithfulness . a simple application of psychological tools cannot faithfully characterize all human-type behaviors .
Approach:	They propose a framework to characterize humanoid behaviors in foundation models . they argue that a simple application of psychological tools cannot faithfully characterize all human-like behaviors .
Outcome:	The proposed framework assesses the faithfulness of results based on reproducibility, internal consistency, and generalizability.

Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models (2023.emnlp-main)

Copied to clipboard

Tim Schott, Daniel Furman, Shreshta Bhat

Multilinguality

Challenge:	LLaMA achieves highest scores in both multilingual and English-only evaluations . foundation models inherit inaccuracies from training corpora, which are propagated downstream to the models that are fine-tuned from them.
Approach:	They propose to produce a 20-language factual association dataset and evaluate 5 models in a multilingual test and benchmark a diverse set of 24 models in an English-only test.
Outcome:	The proposed model achieves highest scores in multilingual and English-only evaluations . however, it has significant limitations in its ability to recall facts in languages other than English, plus difficulties related to the location and gender of fact subjects.

MMA: Cross-Domain Knowledge Integration via Mixture of Multi-Domain Agents (2025.findings-emnlp)

Copied to clipboard

Kehang Jia, Juntao Li, Xiaobo Liang, Yisheng Xiao, Yixuan Yang, Min Zhang

Challenge:	achieving synergistic improvements between generalization and domain specialization remains a challenge in pre-training and post-training.
Approach:	They propose a test-time cross-domain knowledge integration method that integrates general-purpose and domain-specific models to enhance their performance on complex, domainspecific tasks.
Outcome:	The proposed method combines the outputs of general-purpose and domain-specific models to improve their performance on complex, domainspecific tasks.

MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages (2024.emnlp-main)

Copied to clipboard

Marco Gaido, Sara Papi, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, Matteo Negri

Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake? (2023.findings-emnlp)

Challenge:	Existing speech FMs fall short of full compliance with open-source principles . existing models do not have model weights, code, and training data publicly available .
Approach:	They propose to use a CC-BY license to create open-source speech FMs for EU languages . they collect suitable training data by surveying automatic speech recognition datasets .
Outcome:	The proposed model can be used in the 24 official languages of the European Union.

Copied to clipboard

Yuwei Bao, Keunwoo Yu, Yichi Zhang, Shane Storks, Itamar Bar-Yossef, Alex de la Iglesia, Megan Su, Xiao Zheng, Joyce Chai

Representation Potentials of Foundation Models for Multimodal Alignment: A Survey (2025.emnlp-main)

Challenge:	despite advances in AI, it remains a challenge to develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in various tasks.
Approach:	They propose to use a multimodal benchmark dataset to study whether interactive task guidance systems can be quickly adapted to perceptually enabled tasks.
Outcome:	The proposed models demonstrate fair performances in some cases with no training . the results will provide a stepping stone for future work on situated task guidance .

Copied to clipboard

Jianglin Lu, Hailing Wang, Yi Xu, Yizhou Wang, Kuo Yang, Yun Fu

Representation Learning

Challenge:	foundation models learn highly transferable representations through large-scale pretraining on diverse data.
Approach:	They examine the representation potentials of foundation models by examining their latent capacity to capture task-specific information within a single modality while providing a transferable basis for alignment and unification across modalities.
Outcome:	The foundation models exhibit remarkable similarities across architectures and modalities, the authors show . the models can capture task-specific information within a single modality while providing a transferable basis for alignment and unification across modality.

X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity (2023.findings-emnlp)

Copied to clipboard

Taejun Yun, Jinhyeon Kim, Deokyeong Kang, Seonghoon Lim, Jihoon Kim, Taeuk Kim

Cross-Lingual Transfer

Challenge:	Cross-lingual transfer (XLT) is an emergent ability of multilingual language models that preserves their performance when evaluated in non-English languages.
Approach:	They propose to use sub-network similarity between two languages as a proxy for XLT prediction.
Outcome:	The proposed method shows proficiency in ranking candidates for zero-shot XLT, achieving an improvement of 4.6% on average in terms of NDCG@3.

M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models (2024.findings-emnlp)

Copied to clipboard

Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, Arman Cohan

Text Clustering

Challenge:	Existing evaluation benchmarks for foundation models in understanding scientific literature focus on single-document tasks.
Approach:	They propose a multi-modal, multi-document scientific question answering benchmark . it uses expert-annotated questions that span 70 natural language processing paper clusters .
Outcome:	The proposed benchmarks underperform human experts in multi-modal reasoning and retrieval of scientific data.

Distillation Traps and Guards: A Calibration Knob for LLM Distillability (2026.acl-long)

Copied to clipboard

Weixiao Zhan, Yongcheng Jing, Leszek Rutkowski, Dacheng Tao

CRAB: Assessing the Strength of Causal Relationships Between Real-world Events (2023.emnlp-main)

Challenge:	Knowledge distillation (KD) transfers capabilities from large language models (LLMs) to smaller students, yet it can fail unpredictably and also underpins model leakage risks.
Approach:	They propose a method that allows teachers to control their distillability via reinforcement fine-tuning (RFT) they propose to use tail noise, off-policy instability, and the teacher–student gap to improve KD.
Outcome:	The proposed method outperforms SFT and KD baselines and can be used to protect teachers and students from bottlenecks.

Copied to clipboard

Angelika Romanou, Syrielle Montariol, Debjit Paul, Leo Laugier, Karl Aberer, Antoine Bosselut

Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training (2026.acl-long)

Challenge:	Existing models for reasoning about events in narratives do not understand the complexity of the causal relationships of events in the narrative.
Approach:	They propose a Causal Reasoning Assessment Benchmark to evaluate causal understanding of events in narratives.
Outcome:	The proposed model performs worse when models are derived from complex causal structures than simple linear causal chains.

Copied to clipboard

Lei Liu, Hao Zhu, Xiaoyan Yang, Yue Shen, Zhixuan Chu, Jian Wang, Jinjie Gu, Kui Ren

Rethinking Diverse Human Preference Learning through Principal Component Analysis (2025.findings-acl)

Challenge:	Large language models (LLMs) have impressive capabilities across a wide range of domains, but their generalpurpose pre-training objectives often leave them illsuited for specialized applications such as healthcare.
Approach:	They propose a perplexity-aware data scaling law that establishes a predictive relationship between the perplexities of domain-specific data and the test loss.
Outcome:	Experiments on medical and general-domain benchmarks show that the proposed scaling law consistently identifies near-optimal training subsets with significantly reduced data consumption.

Copied to clipboard

Feng Luo, Rui Yang, Hao Sun, Chunyuan Deng, Jiarui Yao, Jingyan Shen, Huan Zhang, Hanjie Chen

Explainability & Interpretability in NLP

Challenge:	Decomposed Reward Models extract diverse human preferences from binary comparisons without fine-grained annotations.
Approach:	They propose a decomposed reward model that extracts diverse human preferences from binary comparisons without fine-grained annotations.
Outcome:	The proposed approach extracts diverse human preferences from binary comparisons without fine-grained annotations.

Beyond Completion: A Foundation Model for General Knowledge Graph Reasoning (2025.findings-acl)

Copied to clipboard

Yin Hua, Zhiqiang Liu, Mingyang Chen, Zheng Fang, Chi Man Wong, Lingxiao Li, Chi Man Vong, Huajun Chen, Wen Zhang

Structured Data in NLP

Challenge:	Existing foundation models for general knowledge graph reasoning have focused on their structural aspects, with most efforts restricted to in-KG tasks.
Approach:	They propose a conditional encoding architecture that bridges the gap between textual and structural modalities, enabling seamless integration.
Outcome:	The proposed model outperforms baseline models on 28 datasets and is generalized to out-of-KG tasks.

Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solving (2025.findings-emnlp)

Copied to clipboard

Shunfeng Zheng, Yudi Zhang, Meng Fang, Zihan Zhang, Zhitan Wu, Mykola Pechenizkiy, Ling Chen

Information Retrieval

Challenge:	a new study examines the potential of retrieval-augmented generation (RAG) with foundation models to enhance expert-level reasoning.
Approach:	They introduce PhoPile, a high-quality multimodal dataset specifically designed for Olympiad-level physics.
Outcome:	The proposed model can be used to solve Olympiad-level physics problems.

Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era (2025.findings-acl)

Copied to clipboard

Dan Oneata, Desmond Elliott, Stella Frank

RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension (2026.acl-long)

Challenge:	danoneata, et al., 2021): human learning and conceptual representation is grounded in sensorimotor experience.
Approach:	They evaluate image encoders and language-only models to learn which attributes are salient to the models.
Outcome:	The proposed models outperform language-only models on attributes predicting extended denser McRae norms and newer Binder datasets.

Copied to clipboard

Yelin Chen, Fanjin Zhang, Suping Sun, Yunhe Pang, Yuanchun Wang, Jian Song, XiaoYan Li, Lei Hou, Shu Zhao, Jie Tang, Juanzi Li

Challenge:	Existing benchmarks for understanding research papers offer limited fine-grained evaluation at scale.
Approach:	They propose a large-scale question-answering benchmark built from review–rebuttal exchanges of high-quality computer science papers.
Outcome:	The proposed model is based on human-verified QA pairs and contains 15K questions.

MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution (2026.acl-long)

Copied to clipboard

Libo Sun, Jiwen Zhang, Siyuan Wang, Zhongyu Wei

Challenge:	Mobile GUI agents powered by large foundation models can perform tasks autonomously, but frequent updates that alter UI appearance and reorganize workflows cause agents trained on historical data to fail.
Approach:	They propose a memory-driven adaptive agent framework with stationary memory that links visual features to stable functional semantics and procedural memory that captures stable task intents across varying workflows.
Outcome:	The proposed framework improves performance over memory-augmented baselines and offline benchmarks on AndroidWorld.

OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization (2025.acl-long)

Copied to clipboard

Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, Dong Yu

NBDESCRIB: A Dataset for Text Description Generation from Tables and Code in Jupyter Notebooks with Guidelines (2025.findings-acl)

Challenge:	Existing studies focus on building text-only agents in synthetic environments where the reward signals are clearly defined.
Approach:	They propose a multimodal web agent that can autonomously conduct real-world exploration and improve itself after each iteration.
Outcome:	The proposed agent improves itself after each iteration, demonstrating strong performance across multiple test sets.

Copied to clipboard

Xuye Liu, Tengfei Ma, Yimu Wang, Fengjie Wang, Jian Zhao

ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities (2025.acl-long)

Challenge:	Existing methods for Jupyter Notebooks focus on generating cell-level descriptions from code snippets or table outputs independently.
Approach:	They propose a task to generate personalized cell-level descriptions using code, tables, and user-written guidelines in Jupyter Notebooks.
Outcome:	The proposed task combines code, tables, and user-written guidelines with personalized descriptions to evaluate the performance of existing models.

Copied to clipboard

Adhiraj Ghosh, Sebastian Dziadzio, Ameya Prabhu, Vishaal Udandarao, Samuel Albanie, Matthias Bethge

MMSciCode: Real-world Evaluation of Multilingual Multi-Discipline Scientific Research Coding (2026.acl-long)

Challenge:	ONEBench enables custom benchmarks for specific capabilities while reusing and aggregating samples.
Approach:	They propose a new paradigm that consolidates individual evaluation datasets into a unified, ever-expanding sample pool.
Outcome:	The proposed model evaluation framework is based on dynamic, sample-level evaluation.

Copied to clipboard

Xue Xia, Zheyuan Yang, Arman Cohan, Yilun Zhao

Programming Languages in NLP

Challenge:	MMSciCode is a benchmark for evaluating foundation models in scientific code generation.
Approach:	They propose a multilingual, multi-discipline benchmark for evaluating foundation models in scientific code generation that integrates domain-specific knowledge with algorithmic reasoning.
Outcome:	The new benchmark is annotated by domain experts and features rigorous quality controls to ensure dataset integrity and authenticity.

SEMMA: A Semantic Aware Knowledge Graph Foundation Model (2025.emnlp-main)

Copied to clipboard

Arvindh Arun, Sumit Kumar, Mojtaba Nayyeri, Bo Xiong, Ponnurangam Kumaraguru, Antonio Vergari, Steffen Staab

ToM-SSI: Evaluating Theory of Mind in Situated Social Interactions (2025.emnlp-main)

Challenge:	Existing Knowledge Graph Foundation Models (KGFMs) rely on graph structure, overlooking the rich semantic signals encoded in textual attributes.
Approach:	They propose a dual-module KGFM that integrates transferable textual semantics alongside structure to generate relation identifiers.
Outcome:	The proposed model outperforms ULTRA and ULtra in fully inductive link prediction in more challenging generalization settings.

Copied to clipboard

Matteo Bortoletto, Constantin Ruhdorfer, Andreas Bulling

Linguistics & Cognitive NLP

Challenge:	Existing Theory of Mind (ToM) benchmarks focus on text-only or dyadic interactions, but to address this gap, we propose ToM-SSI: a new benchmark specifically designed to test ToM capabilities in environments rich with social interactions and spatial dynamics.
Approach:	They propose to use the Sally-Anne test to test ToM capabilities in environments rich in social interactions and spatial dynamics.
Outcome:	The proposed model captures a wider range of social cognition than existing models and demonstrates that existing models are still limited in these new tasks.

AraEval: An Arabic Multi-Task Evaluation Suite for Large Language Models (2025.emnlp-main)

Copied to clipboard

Alhanoof Althnian, Norah A. Alzahrani, Shaykhah Z. Alsubaie, Eman Albilali, Ahmed Abdelali, Nouf M. Alotaibi, M Saiful Bari, Yazeed Alnumay, Abdulhamed Alothaimen, Maryam Saif, Shahad D. Alzaidi, Faisal Abdulrahman Mirza, Yousef Almushayqih, Mohammed Al Saleem, Ghadah Alabduljabbar, Abdulmohsen Al-Thubaity, Areeb Alowisheq, Nora Al-Twairesh

TableVista: Benchmarking Multimodal Table Reasoning under Visual and Structural Complexity (2026.findings-acl)

Challenge:	AraEval is a suite of evaluation tasks designed to assess the advanced knowledge, reasoning, truthfulness, and instruction following capabilities of large language models.
Approach:	They propose to use AraEval to assess the advanced knowledge, reasoning, truthfulness, and instruction following capabilities of large language models in the Arabic context.
Outcome:	The evaluation suite covers a broad spectrum of domains, including science, history, religion, and literature.

Copied to clipboard

Zheyuan Yang, Liqiang Shang, Junjie Chen, Xun Yang, Chenglong Xu, Bo Yuan, Chenyuan Jiao, Yaoru Sun, Yilun Zhao

Towards Robust Mathematical Reasoning (2025.emnlp-main)

Challenge:	TableVista evaluates multimodal table reasoning under visual and structural complexity . current models struggle to maintain reasoning consistency when structural complexity combined with visually integrated presentations.
Approach:	They propose a benchmark for evaluating multimodal table reasoning under visual and structural complexity.
Outcome:	The proposed model performs poorly on visual and structural complexity.

Copied to clipboard

Thang Luong, Dawsen Hwang, Hoang H Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu Hoang Trinh, Quoc V Le, Junehyuk Jung

On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation (2026.findings-acl)

Challenge:	IMO-Bench is a suite of advanced reasoning benchmarks that targets the international mathematical Olympiad level.
Approach:	They propose IMO-Bench, a suite of advanced reasoning benchmarks that targets the level of the international mathematical Olympiad.
Outcome:	IMO-Bench is a suite of advanced reasoning benchmarks that targets the level of the international mathematical Olympiad.

Copied to clipboard

Chan-Jan Hsu, Liang-Hsuan Tseng, Yi-Cheng Lin, Yen-Chun Kuo, Ju-Chieh Chou, Kai-Wei Chang, Hung-yi Lee, Carlos Busso

VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking (2026.acl-long)

Challenge:	Generative spoken language models are often evaluated using global token perplexity, which overlooks fundamental differences between speech and text modalities.
Approach:	They propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity.
Outcome:	The proposed evaluations more faithfully reflect perceived generation quality, as evidenced by stronger correlations with human-rated mean opinion scores (MOS).

Copied to clipboard

Mark Rothermel, Marcus Kornmann, Marcus Rohrbach, Anna Rohrbach

Ethical NLP

Challenge:	Existing benchmarks for evaluating AFC systems are limited in terms of task scope, modalities, domain, language diversity, realism, or coverage of misinformation types.
Approach:	They propose to use Verified Theses and Statements (VeriTaS) to evaluate AFC systems that are static and subject to data leakage as claims enter pretraining corpora.
Outcome:	The proposed system is robust under large-scale pretraining of foundation models and can be updated in the future.

Small Data, Big Noise: Adversarial Training for Robust ParameterEfficient Fine-Tuning (2026.findings-acl)

Copied to clipboard

Eitan Cohen, Idan Simai, Uri Shaham

MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification (2026.findings-acl)

Challenge:	Parameter-Efficient Fine-Tuning (PEFT) is essential for adapting foundation models to downstream tasks, but current methods struggle with robustness to noise and performance degradation on limited training data.
Approach:	They propose a framework that brings adversarial training to PEFT to enhance model robustness and generalization, outperforming alternative approaches.
Outcome:	Experiments with two variants of the proposed framework show that it outperforms existing methods in low-resource settings and under word-level and character-level corruptions.

Copied to clipboard

Bo Zheng, Yudong Chen, Zihua Xiong, Shuai Fang, Peidong He, Yang Yang, Sheng Guo